An empirical evaluation of several test-a-few strategies for testing particular conditions

Existing specification-based testing techniques often generate a comprehensive test suite to cover diverse combinations of test-relevant aspects. Such a test suite can be prohibitively expensive to execute exhaustively due to its large size. A pragmatic strategy often adopted in practice, called test-once, is to identify certain particular conditions from the specification and to test each such condition once only. This strategy is implicitly based on the uniformity assumption that the implementation will process a particular condition uniformly, regardless of other parameters or inputs. As the decision of adopting the test-once strategy is often based on the specification, whether the uniformity assumption actually holds in the implementation needs to be critically assessed, or else the risk of inadequate testing could be non-negligible. As viable alternatives to reduce such a risk, this paper proposes a family of test-a-few strategies for the testing of particular conditions. It further reports two rounds of experiments that evaluate the effectiveness of the test-a-few strategies as compared to test-once. Our experiments (1) provide clear evidence that the uniformity assumption often, but not always, holds, and that the assumption usually fails to hold when the implementation is faulty, (2) demonstrate that all our proposed test-a-few strategies are statistically more reliable than test-once in revealing faulty programs, (3) show that random sampling is already substantially more effective than test-once, and (4) indicate that, compared to other test-a-few strategies under study, choice coverage seems to achieve a better tradeoff between test effort and effectiveness.